{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第八周作业 DenseNet的复现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 描述DenseNet的实现过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DenseNet中心思想是每一层的输出都作为它之后所有层的输入。或者说每一层的输入是它之前所有层的输出。DenseNet在前向传播时，重用特征，加强了特征的传播。反向传播时又减轻了梯度消失的问题。  \n",
    "\n",
    "具体实现过程以DenseNet-121为例：  \n",
    "\n",
    "输入： 224x224的图\n",
    "卷积层： 7x7的卷积核，步长为2，same padding,输出为112x112  \n",
    "池化层： 3x3的最大池化，步长为2，same padding,输出为56x56  \n",
    "DenseBlock1: （1x1的bottleneck+3x3的卷积）总共6个,输出为56x56 \n",
    "Transition1： 1x1的bottleneck+2x2的平均池化，步长2,输出为28x28   \n",
    "DenseBlock2: （1x1的bottleneck+3x3的卷积）总共12个,输出为28x28   \n",
    "Transition2： 1x1的bottleneck+2x2的平均池化，步长2,输出为14x14   \n",
    "DenseBlock3: （1x1的bottleneck+3x3的卷积）总共24个,输出为14x14   \n",
    "Transition3： 1x1的bottleneck+2x2的平均池化，步长2,输出为7x7   \n",
    "DenseBlock4: （1x1的bottleneck+3x3的卷积）总共16个,输出为7x7   \n",
    "分类层： 7x7的平均池化，输出为1x1 （变成特征向量，全连接后softmax分类）  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 对Growth的理解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "growth rate k是每层feature map数的基本单位。growth rate控制着每一层对全局能贡献多少新信息，不过因为每一层都输入了前面所有层的feature map，growth rate无需很大就能得到一个很好的效果。小的growth rate使得我们每层都很窄，减少了需要计算的参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.对稠密链接的理解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "稠密连接的好处在于对feature map的利用更充分了，所以每层可以比较窄，整个网络也无需太深。而且因为每个DenseBlock都有bottleneck和随机dropout，所以也减轻了过拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 补充的代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"Contains a variant of the densenet model definition.\"\"\"\n",
    "\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "slim = tf.contrib.slim\n",
    "\n",
    "\n",
    "def trunc_normal(stddev): return tf.truncated_normal_initializer(stddev=stddev)\n",
    "\n",
    "\n",
    "def bn_act_conv_drp(current, num_outputs, kernel_size, scope='block'):\n",
    "    current = slim.batch_norm(current, scope=scope + '_bn')\n",
    "    current = tf.nn.relu(current)\n",
    "    current = slim.conv2d(current, num_outputs, kernel_size, scope=scope + '_conv')\n",
    "    current = slim.dropout(current, scope=scope + '_dropout')\n",
    "    return current\n",
    "\n",
    "\n",
    "def block(net, layers, growth, scope='block'):\n",
    "    for idx in range(layers):\n",
    "        bottleneck = bn_act_conv_drp(net, 4 * growth, [1, 1],\n",
    "                                     scope=scope + '_conv1x1' + str(idx))\n",
    "        tmp = bn_act_conv_drp(bottleneck, growth, [3, 3],\n",
    "                              scope=scope + '_conv3x3' + str(idx))\n",
    "        net = tf.concat(axis=3, values=[net, tmp])\n",
    "    return net\n",
    "\n",
    "\n",
    "def densenet(images, num_classes=1001, is_training=False,\n",
    "             dropout_keep_prob=0.8,\n",
    "             scope='densenet'):\n",
    "    \"\"\"Creates a variant of the densenet model.\n",
    "\n",
    "      images: A batch of `Tensors` of size [batch_size, height, width, channels].\n",
    "      num_classes: the number of classes in the dataset.\n",
    "      is_training: specifies whether or not we're currently training the model.\n",
    "        This variable will determine the behaviour of the dropout layer.\n",
    "      dropout_keep_prob: the percentage of activation values that are retained.\n",
    "      prediction_fn: a function to get predictions out of logits.\n",
    "      scope: Optional variable_scope.\n",
    "\n",
    "    Returns:\n",
    "      logits: the pre-softmax activations, a tensor of size\n",
    "        [batch_size, `num_classes`]\n",
    "      end_points: a dictionary from components of the network to the corresponding\n",
    "        activation.\n",
    "    \"\"\"\n",
    "    growth = 24\n",
    "    compression_rate = 0.5\n",
    "\n",
    "    def reduce_dim(input_feature):\n",
    "        return int(int(input_feature.shape[-1]) * compression_rate)\n",
    "\n",
    "    end_points = {}\n",
    "\n",
    "    with tf.variable_scope(scope, 'DenseNet', [images, num_classes]):\n",
    "        with slim.arg_scope(bn_drp_scope(is_training=is_training,\n",
    "                                         keep_prob=dropout_keep_prob)) as ssc:\n",
    "            # 原始图像大小为224*224*3\n",
    "            # 初始化卷积操作\n",
    "            scope = 'Conv_7x7'\n",
    "            net = slim.conv2d(images, 2*growth, [7, 7], stride=2, padding='same', scope=scope)\n",
    "            end_points[scope] = net\n",
    "            # 最大池化\n",
    "            scope = 'MaxPool_3x3'\n",
    "            net = slim.max_pool2d(net, [3, 3], stride=2, padding='same', scope=scope)\n",
    "            end_points[scope] = net\n",
    "\n",
    "            # DenseBlock1 \n",
    "            scope = 'DenseBlock1'\n",
    "            net = block(net, 6, growth, scope=scope)\n",
    "            end_points[scope] = net\n",
    "            \n",
    "            # Transition1 \n",
    "            scope = 'Trans1_Conv1x1'\n",
    "            net = bn_act_conv_drp(net, growth, [1, 1], scope=scope)\n",
    "            end_points[scope] = net\n",
    "            scope = 'Trans1_AvgPool2x2'\n",
    "            net = slim.avg_pool2d(net, [2, 2], stride=2, padding='same', scope=scope)\n",
    "            end_points[scope] = net\n",
    "\n",
    "            # DenseBlock2 \n",
    "            scope = 'DenseBlock2'\n",
    "            net =  block(net, 12, growth, scope=scope) \n",
    "            end_points[scope] = net\n",
    "            \n",
    "            # Transition2 \n",
    "            scope = 'Trans2_Conv1x1'                     \n",
    "            net = bn_act_conv_drp(net, growth, [1, 1], scope=scope)\n",
    "            end_points[scope] = net\n",
    "            scope = 'Trans2_AvgPool2x2' \n",
    "            net = slim.avg_pool2d(net, [2, 2], stride=2, padding='same', scope=scope) \n",
    "            end_points[scope] = net\n",
    "\n",
    "            # DenseBlock3 \n",
    "            scope = 'DenseBlock3'\n",
    "            net =  block(net, 24, growth, scope=scope) \n",
    "            end_points[scope] = net\n",
    "            # Transition3 \n",
    "            scope = 'Trans3_Conv1x1'\n",
    "            net = bn_act_conv_drp(net, growth, [1, 1], scope=scope)\n",
    "            end_points[scope] = net\n",
    "            scope = 'Trans3_AvgPool2x2' \n",
    "            net = slim.avg_pool2d(net, [2, 2], stride=2, padding='same', scope=scope) \n",
    "            end_points[scope] = net\n",
    "\n",
    "            # DenseBlock4 \n",
    "            scope = 'DenseBlock4'\n",
    "            net =  block(net, 16, growth, scope=scope) \n",
    "            end_points[scope] = net\n",
    "            \n",
    "            # 全局池化 \n",
    "            scope = 'AvgPool_7x7' \n",
    "            net = slim.avg_pool2d(net, [7, 7], padding='same', scope=scope)  \n",
    "            end_points[scope] = net\n",
    "\n",
    "            # 全连接\n",
    "            scope = 'Flatten'\n",
    "            net =  slim.flatten(net, scope=scope)\n",
    "            end_points[scope] = net\n",
    "            scope = 'Logits'\n",
    "            logits = slim.fully_connected(net, num_classes, activation_fn=None, scope=scope)\n",
    "            end_points[scope] = logits\n",
    "\n",
    "            # softmax分类\n",
    "            scope='Predictions'\n",
    "            end_points[scope] = tf.nn.softmax(logits, name=scope)\n",
    "\n",
    "    return logits, end_points\n",
    "\n",
    "\n",
    "def bn_drp_scope(is_training=True, keep_prob=0.8):\n",
    "    keep_prob = keep_prob if is_training else 1\n",
    "    with slim.arg_scope(\n",
    "        [slim.batch_norm],\n",
    "            scale=True, is_training=is_training, updates_collections=None):\n",
    "        with slim.arg_scope(\n",
    "            [slim.dropout],\n",
    "                is_training=is_training, keep_prob=keep_prob) as bsc:\n",
    "            return bsc\n",
    "\n",
    "\n",
    "def densenet_arg_scope(weight_decay=0.004):\n",
    "    \"\"\"Defines the default densenet argument scope.\n",
    "\n",
    "    Args:\n",
    "      weight_decay: The weight decay to use for regularizing the model.\n",
    "\n",
    "    Returns:\n",
    "      An `arg_scope` to use for the inception v3 model.\n",
    "    \"\"\"\n",
    "    with slim.arg_scope(\n",
    "        [slim.conv2d],\n",
    "        weights_initializer=tf.contrib.layers.variance_scaling_initializer(\n",
    "            factor=2.0, mode='FAN_IN', uniform=False),\n",
    "        activation_fn=None, biases_initializer=None, padding='same',\n",
    "            stride=1) as sc:\n",
    "        return sc\n",
    "\n",
    "\n",
    "densenet.default_image_size = 224\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.在tinymind的log输出："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "2018-07-05 21:01:37.878191: I tensorflow/core/kernels/logging_ops.cc:79] eval/Accuracy[0.00415039062]\n",
    "2018-07-05 21:01:37.878191: I tensorflow/core/kernels/logging_ops.cc:79] eval/Recall_5[0.025390625]\n",
    "\n",
    "这个是CPU运行一个小时的结果。看起来挺凄惨的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行结果地址： https://www.tinymind.com/executions/25hanw27"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:Anaconda3]",
   "language": "python",
   "name": "conda-env-Anaconda3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
